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ABSTRACT: A system for haptic (i.e. kinesthetic and cutaneous) stimulation of 
the hand is described. While our immediate application involves display of speech 
information, a number of other man-machine interface applications may be 
feasible, including force-feedback devices for computer interaction and human 
pattern extraction from multiple datastreams. Our work is motivated by the 
observation that devices for conveying speech to deaf -blind or deaf individuals via 
tactile stimulation have not attained nearly the level of benefit demonstrated by 
some deaf -blind individuals using the Tadoma method of haptic speech reception. 
The Tadoma user places a hand on the talker's face to sense various speech 
articulatory motions. In an attempt to model more closely the information streams 
available via Tadoma, OMAR was developed to stimulate kinesthetic as well as 
tactile receptors, by moving and vibrating fingers in one or two dimensions using 
hard-disk head-positioning actuators. OMAR is being used in experiments 
involving basic haptic perception and supplementation of speechreading with 
haptic codings of speech correlates obtained via x-ray microbeam measurements. 



INTRODUCTION 

Ever since the pioneering efforts of Gault (1924), researchers have attempted to 
develop vibrotactile devices that could allows deaf individuals to fully receive 
speech, or, as a more modest goal, enable deaf individuals to speechread with 
improved comprehension. Various other sensory channels have also been explored 
to aid speech communication, including vision (e.g. using special eyeglasses that 
superimpose colored lights over a talker's face [Upton, 1968]) and audition (via 
electrodes that directly stimulate cochlear auditory nerves). (For a good selection 
of reprints, see Levitt et. al. 1980.) Experiments using vibrotactile devices have 
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evaluated a variety of speech information and codings (Summers 1992; Mason & 
Frost, 1992), ranging from largely unprocessed speech (conveying speech 
"rhythm") to highly processed speech characteristics. Results suggest that the 
information presented and coding scheme used must be carefully matched to the 
receptive and integrative capabilities of the sensory modality, that learning to use 
alternate modalities for speech reception requires extensive training, and that 
performance may be highly dependent on subject, stimuli, and training/testing 
paradigms used. 

While such work has resulted in a number of commercially-available wearable 
devices and sophisticated tactile laboratory instrumentation, these devices have not 
been as effective in conveying or augmenting speech as the non-instrumented 
method of Tadoma (Bernstein, 1992). By placing his or her hand on a talker's 
face, a Tadoma user can sense various physical correlates of speech production 
such as voicing, vocal tract shape, mouth opening, and nasal and oral aspiration 
(Reed et. al., 1992). Skilled users have learned to receive speech at near-normal 
rates with good comprehension. This existence proof for haptic speech reception 
continues to provide considerable encouragement to the tactile speech community 
and has resulted in novel devices such as MIT's artificial talking face for the study 
of Tadoma (Reed et. al., 1985). 

One obvious difference between Tadoma and vibrotactile devices is that Tadoma 
involves stimulation of kinesthetic receptors in addition to cutaneous. In this report 
we describe instrumentation, which we have called OMAR, that was developed for 
the purpose of stimulating both receptor types, primarily for the display of speech 
information. Actuators are used to control the large-scale movement of fingers on 
one hand and to superimpose high-frequency, low-amplitude vibratory signals onto 
these motion trajectories. Currently, OMAR controls motion in the vertical 
direction only, although we have built a prototype channel for two-dimensional 
motion, and have designed the control electronics for two-channel operation. In the 
sections below, we describe OMAR's hardware and software components, and 
planned initial experiments. 

INSTRUMENTATION 

OMAR can be configured for a maximum of 10 channels of one-dimensional 
motion or 5 channels of two-dimensional motion. The actuator, the head- 
positioning (D.C.) motor assembly from Micropolis 1355 hard disk drives, was 
selected based on considerations of torque, displacement, its relatively small 
curvature, and availability on the surplus market. An analog circuit board was 
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developed that implements motor drive electronics and a positional feedback 
control system that uses actuator position as measured by a high-quality linear 
potentiometer (ETI LCP-12A-25). A sample three-channel, one-dimensional 
system is shown in Figure 1. A plate attached to the end of the actuator arm, 
serving as a finger guide, has a maximum vertical displacement of about 9 cm. 
Each actuator can be easily repositioned via a velcro mount. 




Figure 1. Three one-dimensional OMAR actuators. Each actuator arm has a vertical 
displacement of about 9 cm. Attached to each shaft is a linear potentiometer for position 
measurement. 



A digital microprocessor board was developed to process and deliver signals to the 
actuators. It incorporates a 10 or 16 MHz Motorola MC68000P microprocessor, 64 
Kilobytes of random-access memory (RAM), a 16-bit counter, two 16-bit digital- 
to-analog converters (DACs) for driving one or two actuators, and one each 12-bit 
DAC and 12-bit analog-to-digital converter for auxiliary functions. Board decode 
circuitry allows up to 15 boards to be independently accessed by a host computer 
via 16-bit input and output ports and a 6-bit control port. Software modules have 
been developed for controlling the actuator with arbitrary waveforms, either by 
realtime transmission of sample points from the host computer, or by preloading 
samples in the microprocessor's RAM and specifying an output data rate. The 
microprocessor system is also capable of superimposing arbitrary low-level pulses 
onto the larger-scale positional trajectories, primarily for vibratory stimulation of 
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cutaneous receptors. By using the least significant bit in the position samples to 
trigger a pulse rather than superimposing the pulses onto the samples directly, 
stimulus storage requirements and PC-to-microprocessor board bandwidth are 
reduced dramatically. 

Among the more interesting issues addressed during OMAR development was how 
to obtain a large dynamic range while avoiding perceived noise. While kinesthetic 
receptors are capable of sensing motion over the full range of finger extension, 
over a decimeter, some tactile receptor groups can sense vibration with a fraction 
of a micrometer displacement. To obtain a reasonably wide range of motion, a 16- 
bit DAC was used. (However, due to the limited resolution in the position 
potentiometer and analog drive electronics, low-level vibratory signals essentially 
operate open-loop.) Also, a gain control in the drive electronics allows mapping the 
DAC's full range over only a portion of the actuator's displacement, allowing a 
tradeoff between amplitude resolution and maximum displacement. A related issue 
is that highly sensitive receptors may effect perception of quantization steps in the 
low-bandwidth position samples, despite die lowpass filtering (i.e. smoothing) 
inherent in the drive electronics and finger-loaded actuator. Therefore, another 
function of the microprocessor system is data upsampling, by linear interpolation, 
for increasing sampling rate and reducing quantization effects. 

To avoid the possibility of actuator-induced injury, actuator force must be limited. 
An important issue is selection of the force maximum, since insufficient force 
results in mistracking from finger stiffness and muscle tension, whereas too large a 
force may cause injury. We have taken a conservative approach whereby the 
steady-state force limit was set to about 150 grams, a force easily overridden by the 
finger muscles. For comparison, we have measured the weight of a relaxed finger 
at 28 grams. Informal experience has demonstrated that some individuals must 
learn to relax the fingers being manipulated by OMAR in order to avoid 
mistracking. Limit values for transient signals are somewhat higher than those for 
steady-state, allowing the actuator to overcome inertia and to initiate motion 
quickly and cleanly. Current limiting (to about 1 Ampere) is achieved via a light 
bulb (which exhibits a resistance proportional to temperature) in series with the 
actuator. Filament temperature builds up over a finite time interval, allowing the 
larger transient current spikes. The bulb can also be used to indicate to subjects the 
degree to which they are loading the actuator. 

To evaluate OMAR's capabilities to manipulate fingers, measurements were taken 
using an adjustable dashpot to model effects of finger loading. The dashpot was set 
to give the same overall damping as a finger, when a triangular waveform was 
presented. Frequency response curves were obtained for initial peak-to-peak 
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displacements of 0.25, 0.5, 1.25 and 2.0 cm. Response curves were flat to cutoff, 
with 3 dB cutoff frequencies of, respectively, 19, 11.5, 7.5 and 5.5 Hertz. Informal 
experimentation suggests that these cutoff points are sufficiently high that the 
finger's ability to track the actuator is exceeded. 

INITIAL EXPERIMENTS 

Experiments are being carried out to characterize basic perceptual capabilities of 
the haptic system and to evaluate OMAR's potential for conveying speech via the 
haptic channel, as a supplement to lipreading. A first series of studies, performed at 
Indiana University under the direction of Dr. James C. Craig, consists of 
psychophysical evaluations of the device and the haptic system. The detectability 
of tactile stimuli (250-Hz haversine pulses) in the presence and absence of large 
amplitude finger movements will be measured to determine if kinesthetic 
stimulation interferes with tactile stimulation. A second study involves measuring 
the discriminability of finger movements of the same frequency but different 
amplitudes and finger movements of the same amplitude but different frequencies. 
Finally, the extent to which subjects can attend to changes in finger motion with 
three fingers, as compared with a single finger, will be evaluated. 

The results will provide guidance for stimulus design for subsequent experiments 
in haptic presentation of speech. The most critical issue in evaluating OMAR's 
capability for conveying speech is choice of speech information. A point of 
departure, suggested by Tadoma, is to present articulatory position information. As 
there is no reason to believe that Tadoma supplies the optimal coding for haptic 
speech reception, the stimuli can be manipulated so that the subject's hand lies not 
on a virtual face (as in the instrumented Tadoma case), but, for example, along the 
inside of a virtual vocal tract, so that thumb position might correspond to lower lip 
position, elevation of the index finger to upper lip position, middle finger to 
tongue-tip height, fourth finger to tongue blade height, and fifth finger to dorsal 
tongue height. Furthermore, superimposed vibratory signals might be used to 
convey voicing, fundamental frequency, and/or nasality. Fortunately, such 
information is available from x-ray microbeam recordings that have been generated 
by tracing the movement of metal beads placed on various articulators (e.g. 
lower/upper lip, tongue). Several microbeam traces have been converted to OMAR 
stimulus datafiles for initial experiments. These experiments will evaluate the 
ability of subjects to differentiate between patterns of haptic movement 
corresponding closely to movement of a talker's articulators. 
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CONCLUSION 



While the work described here was motivated by a desire to extend our previous 
research in tactile and visual-tactile speech perception to include kinesthesia, 
knowledge about the haptic system may be much more broadly applicable to man- 
machine interfaces. In particular, the computer-human interface has traditionally 
been conceptualized exclusively in terms of vision (with limited auditory 
signalling) and the only role that haptics has played is in manipulation of keyboard 
and pointing devices. Given the impressive capabilities demonstrated by the haptic 
system, might it not be worthwhile to extend the computer-human paradigm to 
include haptic stimulation by the computer? 

Certainly, haptics is an important component of the new "computer-created 
worlds" paradigm, and experiments in force-feedback have been carried out for 
specialized applications such as telerobotics and display of intermolecular forces 
(Brooks 1992). But even the current (computer) graphical user interface paradigm 
might benefit from a haptic stimulation component that could be as simple as 
vibrotactile signalling by the pointing device whenever the cursor is positioned 
over a selectable objects, or as complex as a force -feedback mouse or a 
multichannel haptic display for human extraction of patterns in multiple 
simultaneous datastreams. The research outlined here may give a general indication 
of the capability of the haptic system to receive, from instrumentation, multiple 
streams of data. 
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